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Abstract 


Command  Decision  Aid  Technology  (COMDAT)  is  a  Technology  Demonstrator  Project 
(TDP)  scheduled  to  take  place  during  the  June  2000  to  March  2007  time  frame.  COMDAT 
aims  to  form  the  basis  for  defining  the  mid-life  upgrade  to  the  Command  and  Control  Infor¬ 
mation  System  (C2IS)  of  the  HALIFAX  Class  frigate.  The  overall  TD  program  consists  of 
developing  an  integrated  Maritime  Tactical  Picture  (MTP),  which  is  being  achieved  through 
three  development  cycles.  Defence  Research  &  Development  Canada  (DRDC)  Valcartier  is 
a  partner  in  the  COMDAT  project,  whose  part  of  the  contribution  consists  of  performing 
an  independent  analysis  of  sea  trial  data  to  assess  the  performance  of  the  MSDF  tech¬ 
nology  compared  the  legacy  Command  &  Control  System  (CCS),  conducting  a  sensitivity 
analysis  of  COMDAT  MSDF  parameters  and  algorithms  to  recommend  improvements  for 
COMDAT  subsequent  cycles,  and  providing  scientific  advises  for  Multi-Sensor  Data  Fusion 
(MSDF)  technology  where  required.  This  report  presents  the  work  performed  under  the 
sensitivity  analysis  task.  The  main  objective  of  this  task  consists  of  evaluating  a  candi¬ 
date  alternative  to  the  Jonker,  Volgenant  &  Castanon  (JVC)  association  algorithm,  that 
is  used  by  COMDAT  MSDF.  This  candidate  is  the  Multiple  Hypothesis  Tracking  (MHT) 
association  algorithm,  an  implemented  version  of  which  is  available  in  DRDC  Valcartier’s 
Concept  Analysis  and  Simulation  Environment  for  Automatic  Target  Tracking  and  Identi¬ 
fication  (CASEWTTI)  test-bed.  The  report  presents  a  comparison  of  the  two  algorithms. 
This  comparison  was  motivated  by  a  performance  evaluation  of  COMDAT  MSDF  in  which 
association  performance  was  not  as  good  as  expected. 


Resume 


Command  Decision  Aid  Technology  (COMDAT)  est  un  demonstrateur  technologique  qui 
s’etend  sur  la  periode  de  juin  2000  a  mars  2007.  L’objectif  principal  de  COMDAT  est  de 
fournir  des  resultats/bases  necessaires  a  la  modernisation  du  systeme  de  commandement  et 
controle  des  fregates  canadiennes  de  classe  Halifax.  L’activite  majeure  de  ce  demonstrateur 
technologique,  realisee  en  trois  cycles  de  developpement,  consiste  a  produire  une  image 
tactique  maritime  unique.  Recherche  &  developpement  pour  la  defense  Canada  (RDDC)  - 
Valcartier  est  un  partenaire  cle  dans  le  projet  COMDAT.  Une  partie  de  sa  contribution 
consiste  a  effectuer  une  analyse  independante  des  donnees  des  essais  en  mer.  Le  but  principal 
de  cette  analyse  consiste  a  comparer  la  technologique  de  Fusion  de  Donnees  Multi-Sources 
(FDMS)  utilisee  par  COMDAT  a  la  technologie  utilisee  par  le  systeme  de  commandement 
et  controle  actuel  des  fregates.  La  contribution  de  RDDC  Valcartier  a  COMDAT  comprend 
egalement  une  etude  de  sensibilite  des  parametres/algorithmes  utilises  par  FDMS,  afin  de 
recommander  les  ameliorations  qui  s’imposeraient.  Le  travail  consigne  dans  ce  document 
est  l’analyse  de  sensibilite  des  algorithmes  utilises  par  COMDAT.  L’objectif  principal  de 
cette  activite  consiste  a  evaluer  les  performances  d’une  solution  de  rechange  a  l’algorithme 
dissociation  Jonker,  Volgenant  &  Castanon  (JVC)  utilise  dans  COMDAT.  II  s’agit  de 
l’algorithme  Multiple  Hypothesis  Tracking  (MHT)  dont  une  implantation  est  disponible 
dans  le  banc  d’essais  Concept  Analysis  and  Simulation  Environment  for  Automatic  Target 
Tracking  and  Identification  (CASE_ATTI)  de  RDDC  Valcartier.  Le  rapport  presente  une 
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etude  comparative  des  deux  algorithmes.  Cette  comparaison  a  ete  motivee  par  les  resultats 
de  revaluation  de  la  performance  de  la  technologie  FDMS  utilisee  par  COMDAT  et  celle 
du  systenre  actuel  de  commandenrent  et  controle.  Dans  cette  derniere,  COMDAT  MSDF 
n’a  pas  nrontre  les  performances  en  nratiere  dissociation  prevues  par  la  theorie. 
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Executive  summary 


Performance  evaluation  within  CASE_ATTI  of  MHT  and  JVC 
association  algorithms  for  COMDAT  TD 

A.  Benaskeur,  S.  Yuen  ,  Z.  Triki;  DRDC  Valcartier  TR  2003  -  287;  Defence  R&D  Canada 

-  Valcartier;  May  2007. 

Defence  Research  &  Development  Canada  (DRDC)  Valcartier  is  a  partner  in  the  Command 
Decision  Aid  Technology  (COMDAT)  TDP.  The  main  part  of  its  contribution  consists  of  i) 
performing  an  independent  sea  trial  data  analysis  to  assess  the  performance  of  the  MSDF 
technology  compared  to  the  legacy  Command  &  Control  System  (CCS);  ii)  performing  a 
sensitivity  analysis  of  COMDAT  Multi-Sensor  Data  Fusion  (MSDF)  parameter  sand  algo¬ 
rithms,  in  order  to  make  recommendations  for  any  possible  improvements  for  COMDAT 
subsequent  cycles;  and  iii)  providing  scientific  advises  for  MSDF  technology  to  the  lead  Lab. 
The  current  report  presents  the  results  of  the  work  performed,  at  DRDC  Valcartier,  as  a 
part  of  the  sensitivity  analysis  of  the  fusion  algorithms  used  in  COMDAT  compared  to  the 
ones  implemented  in  DRDC  Valcartier’s  Concept  Analysis  and  Simulation  Environment  for 
Automatic  Target  Tracking  and  Identification  (CASE  JVTTI)  test-bed.  This  analysis  should 
allow  providing  recommendations  for  improvements  of  the  COMDAT  MSDF  engine.  The 
main  objective  of  this  task  consists  of  evaluating  a  candidate  alternative  to  the  Jonker, 
Volgenant  &  Castanon  (JVC)  association  algorithm  that  is  used  by  COMDAT  MSDF.  This 
candidate  is  the  Multiple  Hypothesis  Tracking  (MHT)  association  algorithm,  an  imple¬ 
mented  version  of  which  is  available  in  CASE_ATTI.  This  report  presents  a  comparison 
of  JVC  and  MHT  algorithms,  which  was  motivated  by  the  results  of  the  previously  con¬ 
ducted  performance  evaluation  of  COMDAT  MSDF  and  CCS.  The  latter  showed  a  slight 
superiority  of  the  CCS  over  the  MSDF  in  terms  of  association  performance.  The  herein 
reported  evaluation  of  the  JVC  and  MHT  shows  that,  even  though  it  is  proved  theoretically 
that  MHT  is  the  best  association  algorithm,  the  presented  results  reveal  no  real  advantage, 
with  real  world  data,  of  MHT  over  JVC.  Nevertheless,  the  fact  that  MHT  shows  no  advan¬ 
tage  over  JVC  with  real  data  should  be  regarded  as  tentative  and  needs  more  verification 
through  additional  experimentation.  Behaviour  of  MHT  with  real  world  data  is  not  well 
understood  yet.  More  sensitivity  analysis  is  required  to  gain  a  clear  idea  of  how  the  MHT 
could  work  in  a  practice,  and  this  analysis  left  a  number  of  questions  unanswered.  Rather 
than  investigating  or  envisaging  the  implementation  of  new  complex  algorithms,  such  a 
MHT,  it  would  be  more  judicious  and  safer  to  optimize  the  currently  available  JVC  version. 
Also,  enhancements  of  other  functionalities,  such  the  gating  and  the  estimation,  may  help 
increase  the  performance  of  the  association  algorithm. 
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developpement  pour  la  defence  Canada  -  Valcartier;  mai  2007. 

Recherche  et  developpement  pour  la  defense  Canada  (RDDC)  -  Valcartier  est  un  partenaire 
cle  dans  le  projet  demonstrateur  technologique  Command  Decision  Aid  Technology  (COM¬ 
DAT).  La  partie  pricipale  de  la  contribution  de  RDDC  Valcartier  a  COMDAT  comprend 
trois  volets  :  i)  la  conduite  d’une  analyse  independante  des  donnees  des  essais  en  mer  afin 
de  comparer  la  technologique  de  Fusion  de  Donnees  Multi-Sources  (FDMS),  utilisee  par 
COMDAT,  a  la  technologie  utilisee  par  le  systeme  de  commandement  et  controle  actuel 
des  fregates  canadiennes ;  ii)  realiser  une  etude  de  sensibilite  des  parametres  et/ou  des  al- 
gorithmes  utilises  par  FDMS  de  COMDAT.  Cette  analyse  sert  de  base  pour  recommander 
d’eventuelles  ameliorations  qui  s’avereraient  necessaires ;  et  enfin  iii)  prodiguer  des  conseils 
scientifiques  en  matiere  de  FDMS  selon  les  besoins  du  projet.  Ce  document  presente  les 
resultats  du  travail  effectue  dans  le  cadre  de  l’analyse  de  sensibilite  des  algorithmes  uti¬ 
lises  dans  COMDAT,  comparativement  a  ceux  disponibles  dans  le  banc  d’essais  Concept 
Analysis  and  Simulation  Environment  for  Automatic  Target  Tracking  and  Identification 
(CASE_ATTI)  de  RDDC  Valcartier.  Une  telle  analyse  devrait  rnener  a  des  recommanda- 
tions  pour  l’amelioration  du  systeme  de  fusion  de  donnees  de  COMDAT.  L’objectif  principal 
de  la  tache  consignee  dans  ce  document  consiste  a  evaluer  les  performances  d’une  solution 
de  rechange  a  l’algorithme  d’association  Jonker,  Volgenant  &  Castanon  (JVC)  utilise  dans 
COMDAT.  II  s’agit  de  l’algorithme  Multiple  Hypothesis  Tracking  (MHT),  dont  une  im¬ 
plantation  est  disponible  dans  CASE_ATTI.  Le  rapport  presente  une  etude  comparative  des 
deux  algorithmes.  Cette  comparaison  est  motivee  par  les  resultats  de  1’evaluation  des  per¬ 
formances  de  la  technologie  utilisee  par  COMDAT  et  le  systeme  actuel  de  commandement 
et  controle.  Cette  etude  a  rnontre  une  legere  superiorite  du  systeme  actuel  sur  COMDAT,  en 
matiere  d’association.  La  comparaison  rapportee  dans  le  present  document  rnontre,  quant 
a  elle,  que,  rnalgre  une  superiorite  theorique  prouvee  du  MHT,  les  resultats  obtenus  ne 
revelent  aucun  avantage  reel  d’une  methode  sur  l’autre  avec  des  donnees  experiment  ales 
reelles.  Dans  un  tel  contexte,  le  comportement  de  l’algorithme  du  MHT  est  encore  mal 
compris  et  une  analyse  plus  approfondie  est  requise  afin  de  statuer  sur  ses  reelles  capa- 
cites.  II  en  a  ete  conclu  qu’envisager  l’implantation  d’un  tel  algorithme  dans  COMDAT  est 
premature,  et  qu’il  serait  plus  judicieux  de  consacrer  cet  effort  a  l’optimisation  de  la  ver¬ 
sion  deja  disponible  de  JVC.  Un  effort  serait  egalement  necessaire  pour  l’amelioration  des 
autres  fonctionnalites  qui  font  partie  du  systeme  de  pistage.  Une  telle  amelioration  aurait 
certainement  un  impact  positif  sur  la  qualite  de  l’association. 
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RDDC 

Recherche  &  Developpement  pour  la  Defense  Canada 

RMSE 

Root  Mean  Square  Error 

SDCH 

Standard  Deviation  of  the  Completeness  History 

TACH 

Time  Averaging  of  the  Completeness  History 

TC 

Track  Continuity  (TC) 

TDP 

Technology  Demonstrator  Project 

TP 

Track  Purity 

VOI 

Volume  Of  Interest 

VUE 

Volume  of  Uncertainty  Ellipsoid 
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1  Introduction 


Studies  of  the  Canadian  Navy’s  operational  requirements  for  maritime  Command  and  Con¬ 
trol  (C2)  in  the  2010  time  frame  have  recommended  increased  emphasis  in  the  areas  of 
data  fusion  and  decision  support  to  the  shipboard  Command  Team.  In  response  to  this 
requirement,  Command  Decision  Aid  Technology  (COMDAT),  a  Technology  Demonstra¬ 
tor  Project  (TDP)  taking  place  during  the  June  2000  to  March  2007  time  frame,  aims  to 
form  the  basis  for  defining  a  Multi-Source  Data  Fusion  (MSDF)  capability  for  the  mid-life 
upgrade  to  the  Command  and  Control  Information  System  (C2IS)  of  the  Halifax  Class 
frigates.  The  overall  TDP  program  consists  of  developing  an  integrated  Maritime  Tactical 
Picture  (MTP),  which  is  being  achieved  through  three  development  cycles.  Human  Factors 
Studies  taking  place  in  COMDAT  will  be  producing  recommendations  used  in  building  the 
human  computer  interfaces. 

Defence  Research  &  Development  Canada  (DRDC)  Valcartier  is  a  partner  in  the  COMDAT 
project,  whose  contribution  consists  of  the  following  tasks: 

1.  Functional  architecture  studies; 

2.  Definition  of  system  Measures  Of  Performance  (MOPs)  and  Measures  Of  Effectiveness 
(MOEs); 

3.  The  application  of  Model-Based  Measures  (MBM); 

4.  Sea  trial  data  analysis  to  assess  the  performance  of  the  MSDF  technology  compared 
the  legacy  Command  &  Control  System  (CCS). 

5.  COMDAT  MSDF  parametric  and  algorithmic  sensitivity  analysis.  This  activity  uses 
the  DRDC  Valcartier’s  Concept  Analysis  and  Simulation  Environment  for  Automatic 
Target  Tracking  and  Identification  (CASE_ATTI)  testbed  [1].  This  analysis  is  aimed 
to  make  improvement  recommendations  for  COMDAT  Cycle  III. 

6.  Provide  scientific  advises  for  MSDF  technology. 

The  work  presented  id  this  document  concerns  the  activity  #5.  This  is  about  the  sensitivity 
analysis,  in  the  CASE_ATTI  test-bed,  of  the  fusion  algorithms  equivalent  to  the  ones  used 
in  COMDAT.  This  analysis  is  aimed  to  provide  recommendations  for  the  improvement  of 
the  COMDAT  MSDF  engine.  This  analysis  includes  a  parametric  level  sensitivity  analysis, 
an  algorithmic  level  sensitivity  analysis,  and  where/ when  possible  a  reasoning-path  level 
sensitivity  analysis.  These  three  levels  of  sensitivity  analysis  are  described  below 

1.  Parametric  level  sensitivity  analysis:  compares  COMDAT  fusion  algorithms 
equivalent  to  the  ones  implemented  in  CASE_ATTI.  Parameter  tuning  is  modified 
(optimized)  to  investigate  possible  improvements.  Such  an  analysis  was  performed 
and,  so  far,  the  obtained  results  are  not  very  conclusive.  The  results  are  partially 
documented  in  [2].  Further  work  is  required  for  this  task. 
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2.  Algorithmic  level  sensitivity  analysis:  compares  COMDAT-like  gating,  associa¬ 
tion,  and  fusion  algorithms  with  CASE_ATTI  algorithms  that  performs  similar  tasks. 
Possible  improvements  to  the  limitations  encountered  during  the  data  analysis  pro¬ 
cess  are  investigated.  The  work  reported  in  this  document  falls  under  this  topic.  It 
presents  the  results  of  the  comparison  of  the  Jonker,  Volgenant  &  Castanon  (JVC) 
association  algorithm  [3],  that  is  similar  to  the  one  used  by  COMDAT  MSDF,  with 
Multiple  Hypothesis  Tracking  (MHT)  association  algorithm  [4]  of  CASE_ATTI.  This 
comparison  was  motivated  by  the  results  of  the  performance  evaluation  [5,  6]  of  COM¬ 
DAT  MSDF  and  CCS.  This  analysis  showed  a  slight  superiority  of  the  CCS  over  the 
MSDF  in  terms  of  association  performance1. 

3.  Reasoning-path  level  sensitivity  analysis:  The  reasoning  path  is  viewed  here 
as  a  possible  sequence  of  single  functions  and  processing  to  achieve  a  given  task. 
Investigating  different  reasoning  paths,  for  the  fusion  system  goes  further  than  just 
the  parameter  and/or  algorithm  changes  in  order  to  improve  the  system  performance. 
This  includes  changing  the  assumptions,  models,  etc.  Alternative  reasoning-paths 
are  proposed  to  overcome  limitations  that  cannot  be  handled  by  parameter  tuning 
and  algorithm  replacement.  Due  the  time  and  resource  constraints,  this  task  was  not 
performed. 

This  work  was  performed  during  the  June- August,  2003  time  period. 

This  report  is  organized  as  follows.  An  overview  of  the  scenarios  performed  during  the  Cycle 
I  sea  trials,  the  configuration  parameters  for  the  different  algorithms  used  in  CASE_ATTI, 
and  the  data  used  are  given  in  Section  3.  Section  4  presents  the  set  of  MOPs  used  for  the 
performance  evaluation.  Section  5  gives  the  results  of  this  evaluation  using  the  selected 
MOPs  applied  to  a  sub-set  of  runs.  Concluding  remarks  and  recommendation  are  given  in 
Section  6. 


Nonetheless,  given  the  small  size  of  the  data  sample  used,  any  noticed  differences  in  association  perfor¬ 
mance  are  not  statistically  significant.  A  larger  data  set  is  required  to  confirm  the  noticed  superiority,  if 
there  is  any.  The  work  presented  here  aims  at  finding  an  alternative  to  the  JVC  approach  used  in  COMDAT. 
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2  Algorithmic  Background  Information 


The  foremost  difficulty  in  multiple-target  tracking  applications  is  to  resolve  the  source  data 
origin  uncertainty  issue.  That  is,  one  has  to  determine  from  which  perceived  target,  if  any, 
each  individual  input  data  report  element  originated  before  the  kinematic  and  identification 
fusion  processes  can  be  performed  with  this  data.  The  correlation  procedure,  broadly 
known  as  data  association,  thus  links  input  data  reports  from  single  or  multiple  sources  to 
perceived  individual  physical  platforms  or  entities  (such  as  aircraft,  missiles,  ships,  etc.). 
The  association  may  be  based  upon  kinematic  parameters  (he.,  position,  velocity,  etc.),  non- 
kinematic  target  attributes  ( e.g .,  emitter  characteristics,  target  signature  features,  etc.),  or 
both  simultaneously  [7]. 

Data  association  process  contains  several  steps,  the  most  critical  of  which  is  the  data  as¬ 
signment.  In  general,  this  step  takes  the  output  of  the  gating  function  and  makes  the  final 
assignments.  It  represents  a  more  formal  procedure  than  gating  for  pairing  input  data  to 
system  tracks;  alone,  data  assignment  is  sufficient  to  perform  the  entire  data  association 
task. 

2.1  Data  Assignment 

In  any  multiple-target  tracking  application  where  the  actual  number  of  true  targets  in  the 
environment  is  unknown  a  priori,  there  is,  at  any  given  time,  a  number  of  plausible  ways 
to  partition  the  source  data  into  tracks  and  false  targets  (false  alarms).  The  objective  of 
the  assignment  task  is  to  relate  each  data  element  received  to  a  number  of  possible  sets  of 
data,  each  one  representing  a  hypothesis  to  explain  the  origin  of  the  report  [7]. 

Two  data  assignment  algorithms,  the  Multiple  Hypothesis  Tracking  (MHT)  and  the  Jonker- 
Volgenant-Castanon  (JVC)  approaches,  are  discussed  next  and  compared  throughout  this 
report.  More  background  information  on  the  MHT  and  JVC  algorithms  can  be  found  in 
the  open  literature  [8,  9,  10,  11],  and  won’t  be  presented  in  this  document.  The  more 
specific  variants  and  detailed  characteristics  of  the  algorithms  used  in  this  comparison  are 
well  documented  through  the  CASE_ATTI-related  publications  [12,  13,  1,  14,  7,  15,  16]  and 
documentations  provided  with  the  distribution  CD. 

2.1.1  MHT 

The  Multiple  Hypothesis  Tracking  (MHT)  algorithm  [4]  uses  the  knowledge  that  a  report 
exists  and  compute  the  probability  that  this  report  can  be  associated  to  an  existing  system 
track,  to  a  new  track,  or  to  a  false  alarm.  The  MHT  algorithm  intrinsically  possesses  the 
desirable  feature  of  multiple-scan  correlation.  The  basic  philosophy  behind  the  MHT  is  that 
there  are  many  situations  that  occur  in  the  tracking  of  targets  where  there  is  ambiguous 
information.  Rather  than  using  some  arbitrary  criteria  to  make  a  decision  as  to  how  to 
remove  the  ambiguity  and  resolve  the  difficult  assignments  immediately,  it  is  more  optimal 
to  wait  until  more  input  data  elements  have  been  collected  and  there  is  enough  information 
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data  available  to  substantiate  or  refute  these  assignments  and  thus  postponed  the  final  de¬ 
cision  as  to  the  correct  one.  With  the  MHT  approach,  all  possibilities  concerning  the  origin 
of  received  source  data  are  enumerated  as  alternative  hypotheses.  These  hypotheses  will, 
in  general,  contain  some  groupings  of  input  data  elements  into  tracks  and  the  identifica¬ 
tion  of  other  elements  to  be  false  alarms.  The  probability  of  each  hypothesis  is  computed 
and,  ideally,  all  hypotheses  are  maintained  and  re-evaluated  when  subsequent  source  data 
are  received.  An  hypothesis  whose  probability  is  increased  correspond  to  the  case  in  which 
subsequent  reports  increase  the  likelihood  of  these  data  associations  [7] .  While  it  takes  time 
for  the  system  to  select  the  correct  hypothesis,  if  a  decision  is  required  earlier,  the  highest 
probability  hypothesis  would  be  used,  and  under  no  circumstances  would  this  hypothesis 
ever  be  any  worse  than  what  any  other  algorithm  would  provide.  The  only  real  disadvan¬ 
tage  of  the  MHT  algorithm  over  the  others  is  its  processing  load.  This  is  discussed  in  the 
next  subsection. 

The  computational  requirements  necessitated  by  the  ability  to  retain  multiple  interpreta¬ 
tions  of  the  situation  represent  the  main  drawback  of  the  MHT  algorithm  (and  also  the 
principal  deterrent  to  its  widespread  usage).  Allowing  for  multiple  targets  and  for  new 
track  initiation  can  lead  to  complex  bookkeeping  and  to  a  rapid  growth  in  the  number  of 
hypotheses  formed.  Without  appropriate  control  mechanisms,  the  number  of  hypotheses 
is  exponentially  increasing  with  time  [7].  Since  it  is  impractical  to  maintain  all  hypothe¬ 
ses,  a  suboptimal  implementation  which  includes  mechanisms  for  keeping  the  number  of 
hypotheses  down  must  be  used.  Indeed,  the  success  of  an  efficient  MHT  implementation  is 
primarily  dependent  on 

1.  the  development  of  a  scheme  which  can  effectively  limit  the  number  of  hypotheses 
formed, 

2.  the  development  of  methods  of  reducing  the  large  number  of  hypotheses  ultimately 
formed  to  a  manageable  number  of  hypotheses,  and, 

3.  the  construction  of  an  efficient  data  structure  to  represent  the  hypothesis  tree  and 
track  data  information. 

Clustering  is  a  practical  approach  that  can  be  combined  with  MHT  to  reduce  its  compu¬ 
tational  requirements.  Clustering  is  the  process  of  dividing  the  entire  set  of  system  tracks 
and  input  data  elements  into  independent  groups  (or  clusters).  Instead  of  solving  one  large 
problem,  a  number  of  smaller  problems  are  solved  in  parallel. 

2.1.2  JVC 

The  JVC  (Jonker-Volgenant-Castanon)  is  one  the  set  algorithms  that  optimally  solve  (under 
constraints)  the  2-D  assignment  algorithm.  The  JVC  algorithm  solves  this  constrained 
optimization  into  two  stages.  The  first  stage  ensures  the  feasibility  of  the  assignment 
problem  by  an  appropriate  conditioning  of  the  assignment  matrix  and  is  similar  to  the 
auction  algorithm.  The  second  stage  is  similar  to  a  sparse  version  of  the  Munkres  algorithm. 
It  considers  only  the  finite  values  of  the  assignment  matrix  (prohibited  associations  are 
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represented  in  the  assignment  matrix  as  very  large  values).  In  addition  to  ensuring  the 
feasibility  of  the  assignment  problem,  the  initialization  phase  of  the  algorithm  provides 
a  very  good  conditioning  for  the  paths  search  and  makes  the  JVC  faster  than  the  other 
shortest  path  algorithms.  By  an  appropriate  increase  of  the  overhead,  the  algorithm  reduces 
the  required  computations  in  the  search  step  and  by  the  same  way  the  overall  computation 
time. 


Besides  data  association,  the  filtering  is  also  very  important  in  tracking  system.  Neverthe¬ 
less,  tracking  performance  with  even  the  best  designed  filter  may  become  very  degraded  in 
the  presence  of  miscorrelation.  The  effects  of  miscorrelation  can  completely  invalidate  the 
the  used  filter  and  lead  to  divergence.  The  Interacting  Multiple  Model  (IMM)-based  filter¬ 
ing  technique,  used  by  both  COMDAT  MSDF  and  CASE_ATTI  to  overcome  the  limitations 
of  dynamics  modeling,  is  briefly  discussed  for  completeness. 

2.2  IMM 

Some  practical  model  of  target  motion  is  assumed  for  the  design  of  the  Kalman  filter.  This 
target  kinematics  model  is  generally  simple  (such  as  a  straight-line  path,  a  slow  turn,  a 
sharp  turn,  etc.)  and  assumed  to  be  described  by  well-known  physical  laws  ( e.g .,  ballis¬ 
tic  laws,  etc.).  The  target  dynamics  are  modeled  through  the  use  of  continuous  random 
variables  statistically  described  by  known  parameters.  The  Kalman  filter  will  provide  op¬ 
timum  estimates  of  target  position  and  velocity  (ie.,  it  will  minimize  the  mean-squared 
error)  only  if  the  underlying  target  model  is  correct.  Unpredictable  changes  to  the  assumed 
target  motion  model  are  called  maneuvers.  Any  mismatch,  during  a  maneuver,  between 
the  actual  kinematics  behavior  of  a  target  and  the  motion  model  assumed  for  filter  design 
can  completely  degrade  the  performance  of  the  estimation  technique  (ie.,  mean  tracking 
errors  will  develop). 

At  the  moment  when  the  target  maneuver  begins,  there  may  be  a  step  discontinuity  in 
acceleration.  The  target  acceleration  can  be  well  modeled  as  a  continuous  random  variable, 
both  before  and  after  the  maneuver  event,  but  the  step  acceleration  input  is  not  efficiently 
handled  by  the  continuous  model.  Unless  this  type  of  acceleration  is  accounted  for,  the 
resulting  time  lag  between  the  initiation  and  the  detection  of  the  acceleration  change  can 
lead  to  track  loss.  In  the  presence  of  a  high  speed  maneuvering  target,  with  noise  and 
clutter,  it  is  critical  to  be  able  to  detect  the  maneuver  and  continue  tracking  the  target  in 
its  new  course  as  quickly  as  possible.  Using  different  filtering  techniques,  there  are  various 
approaches  proposed  in  the  literature  for  handling  target  maneuvers.  The  resulting  filter  is 
often  called  an  adaptive  Kalman  filter.  One  of  the  widely  used  adaptive  Kalman  filtering 
approach  that  gives  very  good  tracking  performance  is  based  on  the  use  of  Interacting 
Multiple  Model  (IMM)  Kalman  filters. 

In  the  IMM  approach,  the  state  estimate  is  computed  at  time  k  under  each  possible  current 
model  using  r  filters,  with  each  filter  using  a  different  combination  of  the  model-conditioned 
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estimates  ( i.e .,  mixed  initial  condition).  The  approximation  is  that  the  past  through  A;  —  1 
is  summarized  by  r  model-conditioned  estimates  and  covariances.  The  input  to  the  filter 
matched  to  model  j  is  obtained  from  an  interaction  of  the  r  filters,  which  consists  of  the 
mixing  of  the  estimates  at  time  A:  —  1  with  the  weights  fij  (probabilities)  called  the  mixing 
probabilities. 

2.3  Parameters 

The  different  parameters  and  algorithms  used  for  the  tuning  and  the  optimization  of  the 
performance  of  CASE_ATTI  are  list  below. 

1.  Gating:  uses  ellipsoidal  gating,  with  two  parameters  to  tune 

(a)  Gating  threshold  (ellipse  area) 

(b)  Time  aligner’s  process  noise,  for  manoeuvring  targets 

(c)  Maximum  velocity 

2.  Assignation:  uses  MHT  (with  and  without  clustering)  and  JVC.  The  following  pa¬ 
rameters  were  used  for  the  optimization 

(a)  Number  of  hypotheses  for  the  MHT  tree 

(b)  Number  of  best  nodes  to  keep  for  MHT  tree 

(c)  Time  aligner’s  process  noise  for  manoeuvring  targets 

3.  Filtering:  uses  Interactive  Multiple  Model  (IMM)  Kalman  Filter  [17]  with  following 
parameters 

(a)  Number  of  sub-filters 

(b)  Process  noise  attributed  to  the  different  sub-filters 

4.  Track  Management:  uses  the  following  parameter 

(a)  Track  confirmation.  To  confirm  a  track,  a  N  number  of  hits  out  of  a  M  number  of 
attempts  are  required.  The  default  value  in  CASEWTTI  is  3  out  of  5.  However, 
that  is  not  necessarily  a  suitable  value  that  give  good  performance,  thus  the 
value  is  adapted  to  each  run. 

5.  Output  Manager 

(a)  JVC:  Basic  output  manager  with  all  confirmed  track  output  strategy 

(b)  MHT:  Hard  Soft  Prediction  (HSP)  output  manager  with  newly  confirmed  hard 
decided  track  output  strategy 


6 


DRDC  Valcartier  TR  2003  -  287 


3  Scenarios,  data  generation,  and  algorithm  behavior 


The  data  used  by  DRDC  Valcartier  for  the  herein  reported  algorithmic  analysis  is  the  same 
as  the  one  used  in  the  previously  performed  sea  trial  data  analysis  [5,  6].  This  data  set  was 
provided  to  DRDC  Valcartier  by  Lockheed  Martin  Canada  (LMC),  through  DRDC  Atlantic, 
in  a  confidential  CD  labelled  “MOPs  MSDF,  Volume  C.l”.  DRDC  Valcartier  received  two 
CDs  containing  COMDAT  sea  trial  data.  The  two  data  sets  present  small  differences,  due 
mainly  to  the  time  reference  problem.  Therefore,  the  performed  comparison  is  based  on 
the  same  COMDAT  Cycle  I  sea  trial  scenarios.  Note  that  the  available  COMDAT  data  set 
includes  radar  data  (both  SPS-49  and  SG-150),  IFF  data  (both  SPS-49  and  SG-150)  and 
ESM  data.  For  the  purpose  of  this  analysis,  only  the  radar  data  was  processed. 

3.1  Methodology 

The  approach  adopted  to  compare  JVC  to  MHT  is  to  use  implementations  of  both  algo¬ 
rithms  within  CASE_ATTI  environment.  COMDAT  Cycle  I  sea  trial  data  is  then  pro¬ 
cessed  through  both  sets  of  algorithms.  Also,  it  is  worth  noting  that  the  implementation 
of  JVC/IMM  algorithms  within  CASE_ATTI  is  different  from  their  implementation  within 
COMDAT,  even  though  the  approach  remains  the  same.  This  approach  is  the  best  way  to 
compare  the  association  algorithms  because  it  allowed  other  data  fusion  system  components 
(gating,  estimation,  track  maintenance,...)  to  be  held  constant. 

The  ground  truth  information  and  the  sensor  reports  are  used  as  inputs  to  CASE_ATTI. 
In  order  to  keep  this  document  UNCLASSIFIED,  no  information  from  the  sensor  will  be 
presented.  Only  the  track  information  generated  by  CASE_ATTI  will  be  used  for  the  com¬ 
parison.  During  the  preliminary  experimentations  with  different  runs,  it  was  realized  that 
some  of  the  data,  namely  the  contacts  and  the  ground  truth,  are  unsynchronized  and  biased. 
Also,  some  of  the  ground  truth  data  are  missing  from  the  provided  data  CD.  Therefore, 
appropriate  adjustments  were  brought  to  apply  correction  where  necessary.  However,  noth¬ 
ing  could  be  done  for  the  missing  data,  thus  some  of  the  runs  are  not  analyzed.  This 
concerns  mainly  runs  5,  6  and  10.  Also,  using  CASE^VTTI  test-bed  for  comparison  implies 
the  conversion  of  COMDAT  ground  truth  and  measured  data  into  CASE_ATTI  ground 
truth  and  measured  data  format.  Modifications  were  brought  to  CASE_ATTI  to  render 
this  conversion  possible.  What  also  was  retained  from  the  experimentations  performed  is 
that  the  implementation  of  MHT  in  CASE_ATTI  does  not  allow  tracking  when  only  the 
SPS-49  radar  system  reports.  MHT  will  work  correctly  only  where  one  has  both  SPS-49 
and  SG-150  reports.  Therefore,  only  such  sections  are  used  for  the  comparison  of  JVC  and 
MHT.  Also,  the  clustering  within  MHT  does  not  necessarily  yield  a  good  performance  for 
most  of  the  runs.  A  poor  clustering  performance  will  increase  the  computation  burden  due 
the  overhead  required  by  the  cluster  management  (merge  and  split)  operations. 

3.2  Sea  Trials 

The  sea  trials  were  conducted  during  Spring  2001  based  on  13  scenarios.  These  scenarios 
are  briefly  described  in  the  current  section,  and  the  corresponding  ground  truth  information 
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presented.  This  information  and  the  sensor  reports  are  used  as  inputs  to  the  CASE_ATTI 
test-bed.  The  following  sections  give  a  summary  of  the  scenarios,  data,  run  properties,  and 
the  algorithm  behaviour.  Note  that  each  one  of  these  runs  has  been  experimented  with  dif¬ 
ferent  parameters  and  algorithms  in  CASE_ATTI.  In  total,  more  than  700  experimentations 
were  conducted. 

Tables  1  to  7  below  give,  for  both  JVC  and  MHT,  the  combination  of  parameters  that  gave 
the  best  performance  for  the  considered  runs.  Note  that,  during  the  algorithm-tuning  phase, 
all  the  listed  parameters  were  varied.  Given  the  very  limited  time  available  to  perform  the 
analysis,  no  procedure  was  developed;  rather,  an  ad  hoc  approach  was  used.  Also,  the 
parameters  were  optimized  for  each  run  separately. 

3.2.1  Run  1 

The  scenario  of  Run  1  is  described  on  Figure  1.  Both  aircraft  begin  flying  as  close  together 
as  safety  allows,  on  the  same  course,  elevation  and  speed.  At  15nm  one  aircraft  descends 
to  200 ft.,  at  5 nm  the  run  ends. 


S80  Ground  Truth  3D 
S546  Ground  Truth  3D 
S80  Ground  Truth  2D 
S546  Ground  Truth  2D 


(b)  Ground  Truth  data 


Figure  1:  Run  1 


Algorithm  behavior 

Tracking  of  both  targets  was  possible  with  JVC  during  the  whole  run.  For  the  IMM,  four 
filters  were  used  instead  of  three.  The  latter  configuration  makes  the  system  loose  the 
targets.  It  was  impossible  to  track  both  targets  during  the  whole  run  with  the  MHT.  One 
track  is  created  at  the  beginning  of  the  run  and  remains  until  the  end,  while  the  other 
track  is  only  created  at  the  end  of  the  run.  MHT  without  clustering  often  causes  the  re¬ 
initialization  (out  of  hypotheses)  of  the  tracking.  Therefore  only  the  MHT  with  clustering 
is  considered  for  this  run.  Tracking  of  both  targets  was  also  possible,  for  Run  IB,  with 
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JVC  during  the  whole  run.  Tracking  of  both  targets  was  possible  during  the  whole  run  with 
MHT  using  an  IMM  (with  three  filters).  The  clustering  is  not  used  for  this  run  because: 

1.  it  takes  too  long  time  to  run  (more  than  an  hour  for  one  step),  or 


2.  it  reduces  the  quality  of  tracks  compared  to  the  MHT  without  clustering. 


Parameters 

Values  JVC 

Value  MHT 

Maximum  Velocity 

500  m/s 

500  m/s 

Coordinate  Converter 

Standard  mode 

Standard  mode 

Gating 

Ellipsoidal  Gate 

Ellipsoidal  Gate 

Gate  Probability 

0.999 

0.999 

Gate  Process  Noise 

400 

400 

Filter 

IMM  4 

IMM  4 

IMM  Process  Noise 

5,  100,  200,  400 

5,  100,  200,  400 

M  out  Of  N  System  Track 

3/10 

3/5 

Assignation 

JVCNN 

MHT  Clustering 

JVCNN  Process  Noise 

400.0 

MHT  Process  Noise 

400.0 

Mode  of  pruning 

Best  node 

Number  of  best  node  to  keep 

5 

Number  of  hypotheses 

100 

SPS49  Scan_RPM  &  Scan_bearing_division 

12  &  4 

12  &  4 

SG150  Scan_RPM  &  Scan_bearing_division 

60  &  32 

60  &  32 

Output 

Basic  Output 

HSP  Output 

Table  1:  Parameter  and  values  of  Run  1 


3.2.2  Run  2 

The  scenario  of  Run  2  is  described  on  Figure  2.  Both  aircraft  begin  flying  as  close  to¬ 
gether  as  safety  allows,  on  the  same  course,  elevation  and  speed.  At  15nm  one  aircraft  will 
accelerate  to  250 mph.  At  5 nm  the  run  ends. 

Algorithm  behaviour 

Tracking  of  both  targets  was  possible  with  JVC  during  the  whole  run.  Tracking  of  both 
targets  was  possible  only  on  a  portion  of  the  run  with  MHT.  Some  ad  hoc  tunings  allow 
tracking  both  targets  during  the  whole  run  (requires  more  investigation).  The  clustering  is 
not  used  since  it  reduces  the  quality  of  tracks. 

3.2.3  Run  3 

The  scenario  of  Run  3  is  described  on  Figure  3.  The  two  aircraft  fly  the  same  profile,  but 
following  opposite  S  shape,  perform  a  mean  line  of  advance  toward  the  ship,  while  each 
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Parameters 

Values  JVC 

Value  MHT 

Maximum  Velocity 

500  m/s 

500  m/s 

Coordinate  Converter 

Standard  mode 

Standard  mode 

Gating 

Ellipsoidal  Gate 

Ellipsoidal  Gate 

Gate  Probability 

0.999 

0.999 

Gate  Process  Noise 

50 

50 

Filter 

IMM  4 

IMM  3 

IMM  Process  Noise 

5,  10,  30,  50 

5,  25,  50 

M  out  Of  N  System  Track 

3/10 

3/5 

Assignation 

JVCNN 

MHT 

JVCNN  Process  Noise 

50.0 

MHT  Process  Noise 

50.0 

Mode  of  pruning 

Best  node 

Number  of  best  node  to  keep 

30 

Number  of  hypotheses 

150 

SPS49  Scan RPM  &  Scan bearing division 

12  &  4 

12  &  4 

SG150  Scan RPM  &  Scan bearing division 

60  &  32 

60  &  32 

Output 

Basic  Output 

HSP  Output 

Table  2:  Parameter  values  for  Run  lb 


S80  Ground  Truth  3D 
S546  Ground  Truth  3D 
S80  Ground  Truth  2D 
S546  Ground  Truth  2D 


(b)  Ground  Truth  data 


Figure  2:  Run  2 
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Parameters 

Values  JVC 

Value  MHT 

Maximum  Velocity 

500  m/s 

500  m/s 

Coordinate  Converter 

Standard  mode 

Standard  mode 

Gating 

Ellipsoidal  Gate 

Ellipsoidal  Gate 

Gate  Probability 

0.999 

0.999 

Gate  Process  Noise 

75.0 

75.0 

Filter 

IMM  3 

IMM  3 

IMM  Process  Noise 

5,  20,  75 

5,  20,  75 

M  out  Of  N  System  Track 

3/10 

3/5 

Assignation 

JVCNN 

MHT 

JVCNN  Process  Noise 

75.0 

MHT  Process  Noise 

75.0 

Mode  of  pruning 

Best  node 

Number  of  best  node  to  keep 

5 

Number  of  hypotheses 

100 

SPS49  Scan  RPM  &  Scan  bearing  division 

12  &  4 

12  &  4 

SG150  Scan  RPM  &  Scan  bearing  division 

60  &  32 

60  &  32 

Output 

Basic  Output 

HSP  Output 

Table  3:  Parameter  values  for  Run  2 


aircraft  weaves  across  the  mean  line  of  advance.  The  maximum  separation  between  aircraft 
is  initially  3 nm.  The  maximum  separation  diminishes  as  it  approaches  the  ship  ( i.e .,  3nm 
at  60nm  range,  1.5  at  30nm  range,  etc.).  Safety  can  be  assured  by  altitude  separation. 


S80  Ground  Truth  3D 
S546  Ground  Truth  3D 
S80  Ground  Truth  2D 
S546  Ground  Truth  2D 


Figure  3:  Run  3 
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Algorithm  behaviour 

Tracking  of  both  targets,  using  JVC,  was  possible  during  the  whole  run,  but  one  track  has 
split  into  two  tracks.  It  was  impossible  to  track  both  targets  during  the  whole  run,  using 
MHT,  because  most  of  time  there  were  only  SPS-49  reports.  The  clustering  is  not  used  for 
this  run  because  it  reduces  the  quality  of  tracks. 


Parameters 

Values  JVC 

Values  MHT 

Maximum  Velocity 

500  m/s 

500  m/s 

Coordinate  Converter 

Standard  mode 

Standard  mode 

Gating 

Ellipsoidal  Gate 

Ellipsoidal  Gate 

Gate  Probability 

0.999 

0.999 

Gate  Process  Noise 

500.0 

500.0 

Filter 

IMM  4 

IMM  4 

IMM  Process  Noise 

5,  200,  400,  500 

5,  200,  400,  500 

M  out  Of  N  System  Track 

3/15 

3/5 

Assignation 

JVCNN 

MHT 

JVCNN  Process  Noise 

500.0 

MHT  Process  Noise 

500.0 

Mode  of  pruning 

Best  node 

Number  of  best  node  to  keep 

10 

Number  of  hypotheses 

50 

SPS49  Scan  RPM  &  Scan  bearing  division 

12  &  4 

12  &  4 

SG150  Scan  RPM  &  Scan  bearing  division 

60  &  32 

60  &  32 

Output 

Basic  Output 

HSP  Output 

Table  4:  Parameter  values  of  Run  3 


3.2.4  Run  4 

For  the  scenario  of  Run  4  (see  Figure  4),  the  two  aircraft  begin  by  flying  towards  one 
another  at  a  shallow  crossing  angle.  At  15 nm  they  cross,  as  close  to  one  another  as  safety 
allows,  separate  and  continue  to  fly  past  the  ship. 

Algorithm  behaviour 

Tracking  of  both  targets,  using  JVC,  was  possible  during  the  whole  run.  Tracking,  using 
MHT,  was  possible  whenever  the  SG-150  starts  to  report.  The  clustering  is  not  used  because 
it  reduces  the  quality  of  tracks,  and  may,  in  some  situations,  not  produce  tracks  at  all. 

3.2.5  Run  5 

Figure  5  shows  Run  5  that  starts  with  a  distance  separation  of  3 miles.  Both  aircraft  fly 
towards  each  other  at  an  acute  angle  of  approach.  They  merge  at  15 nm  from  the  ship  and 


12 


DRDC  Valcartier  TR  2003  -  287 


25  nm 


S80  Ground  Truth  3D 
S546  Ground  Truth  3D 
S80  Ground  Truth  2D 
S546  Ground  Truth  2D 


(b)  Ground  Truth  data 


Figure  4:  Run  4 


stay  as  close  as  safety  allows  for  5 nm.  At  this  point  (lOnm),  they  separate  and  pass  the 
ship  on  either  side.  This  Run  uses  all  IFF  settings. 


25  nm 


T1  Ground  Truth  3D 
T2  Ground  Truth  3D 
T1  Ground  Truth  2D 
T2  Ground  Truth  2D 


(b)  Ground  Truth  data 


Figure  5:  Run  5 


Algorithm  behaviour 

This  run  was  not  analyzed,  since  the  ground  truth  information  is  incomplete  and  data  are 
not  synchronized. 
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3.2.6  Run  6 


Run  6  (see  Figure  6)  starts  with  a  distance  separation  of  3 miles.  Both  aircraft  fly  towards 
each  other  at  an  acute  angle  of  approach.  They  merge  at  15nm  from  the  ship  and  stay  as 
close  as  safety  allows  for  5 nm.  At  this  point  (10n?n),  they  separate  and  pass  the  ship  on 
either  side.  This  Run  is  conducted  without  IFF  settings. 


25  nm 


Figure  6:  Run  6 


Algorithm  behaviour 

As  for  the  Run  5,  this  run  was  not  analyzed,  since  the  ground  truth  information  is  incomplete 
and  data  are  not  synchronized. 

3.2.7  Runs  7  &  8 

During  Run  7,  as  shown  on  Figure  7  (a),  aircraft  begins  at  an  altitude  of  2000/f.  At 
5 nm,  aircraft  descends  rapidly  to  200/f.  Once  past  the  ship  the  aircraft  is  free  to  return 
to  altitude.  For  Run  8,  shown  on  Figure  7  (b),  aircraft  fly  towards  ship.  At  5 nm,  aircraft 
performs  a  90°  turn,  either  left  or  right 

Algorithm  behaviour 

Tracking  of  both  targets,  using  JVC,  was  possible  during  the  whole  run.  Tracking  of  both 
targets  was  possible  during  the  whole  run  using  MHT.  Clustering  works  correctly,  the  track 
quality  is  comparable  to  the  one  given  by  MHT  without  clustering. 

3.2.8  Run  9 

For  Run  9  (see  Figure  8),  both  aircraft  are  to  fly  one  complete  circle  around  the  ship  at  a 
range  of  20 nm.  One  aircraft  flies  in  clockwise  direction,  the  other  flies  counter  clockwise. 
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(b)  Scenario  8 


T1  Ground  Truth  3D 
T2  Ground  Truth  3D 
T1  Ground  Truth  2D 
T2  Ground  Truth  2D 


(c)  Ground  Truth  data 


Figure  7:  Runs  7  &  8 


Algorithm  behaviour 

Tracking,  using  JVC,  of  both  targets  was  possible  during  the  whole  run.  Tracking  of  both 
targets  during  the  whole  run  was  also  possible  using  MHT. 


3.2.9  Run  10 

As  shown  Figure  9,  two  groups  of  targets  approach  the  ownship,  in  Run  10.  Group  1, 
formed  by  T-33  Alpha,  Bravo  and  Charlie  and  Lear  Jet  Alpha,  approaches  on  a  direct  line 
of  constant  bearing.  Lear  jet  Bravo  approaches  the  ownship  on  a  curved  approach.  As  Lear 
Jet  Bravo  crosses  over  the  path  of  the  other  group,  T-33  Alpha  and  Bravo  should  curve  to 
the  same  bearing  and  approach  pattern  as  the  Lear  jet  Bravo. 
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Parameters 

Values  JVC 

Values  MHT 

Values  MHTC 

Maximum  Velocity 

500  m/s 

500  m/s 

500  m/s 

Coordinate  Converter  mode 

Standard 

Standard 

Standard 

Gating 

Ellipsoidal 

Ellipsoidal 

Ellipsoidal 

Gate  Probability 

0.999 

0.999 

0.999 

Gate  Process  Noise 

800.0 

800.0 

800.0 

Filter 

IMM  3 

IMM  3 

IMM  3 

IMM  Process  Noise 

5,  400,  800 

5,  400,  800 

5,  400,  800 

M  out  Of  N  System  Track 

3/5 

3/5 

3/5 

Assignation 

JVCNN 

MHT 

MHT  (Oust.) 

JVCNN  Process  Noise 

800.0 

MHT  Process  Noise 

800.0 

800.0 

Mode  of  pruning 

Best  node 

Best  node 

Number  of  best  node  to  keep 

7 

3 

Number  of  hypotheses 

100 

20 

SPS49  Scan  RPM  &  Bearing  division 

12  &  4 

12  &  4 

12  &  4 

SG150  Scan  RPM  &  Bearing  division 

60  &  32 

60  &  32 

60  &  32 

Output 

Basic 

HSP 

HSP 

Table  5:  Parameter  values  for  Run  7&:8 


Figure  8:  Run  9 


Algorithm  behaviour 

This  run  was  not  analyzed,  since  the  ground  truth  information  is  incomplete  and  data  are 
not  synchronized. 
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Parameters 

Values  JVC 

Values  MHT 

Maximum  Velocity 

500  m/s 

500  m/s 

Coordinate  Converter 

Standard  mode 

Standard  mode 

Gating 

Ellipsoidal  Gate 

Ellipsoidal  Gate 

Gate  Probability 

0.999 

0.999 

Gate  Process  Noise 

50.0 

240 

Filter 

IMM  3 

IMM  3 

IMM  Process  Noise 

5,  160,  240 

5,  160,  240 

M  out  Of  N  System  Track 

3/5 

3/5 

Assignation 

JVCNN 

MHT 

JVCNN  Process  Noise 

240.0 

MHT  Process  Noise 

240.0 

Mode  of  pruning 

Depth  control 

Number  of  level  to  keep 

5 

Number  of  hypotheses 

100 

SPS49  Scan  RPM  &  Bearing  division 

12  &  4 

12  &  4 

SG150  Scan  RPM  &  Bearing  division 

60  &  32 

60  &  32 

Output 

Basic 

HSP 

Table  6:  Parameter  values  for  Run  9 


3.2.10  Run  11 

During  Run  11  (see  Figure  10),  four  targets  approach  the  ownship,  starting  with  T-33 
Alpha  on  the  left,  the  next  is  Lear  Jet  Alpha,  T-33  Bravo  and  then  Lear  Jet  Bravo.  Two 
outer  aircraft  cross  over  and  then  continue  on  the  same  course  as  opposite  track.  Have  the 
two  inner  aircraft  cross  over  and  then  continue  on  the  same  course  as  the  opposite  track. 
T-33  Charlie  flies  in  a  reciprocal  healing  from  the  other  aircraft.  Once  T-33  Charlie  has 
passed  over  the  ownship,  it  may  turn  and  proceed  to  base. 

Algorithm  behaviour 

The  system  is  able  to  track  with  JVC  and  an  IMM  (4  filters).  Tracks  are  only  clear  from  the 
middle  of  the  run.  It  was  hard  to  obtain  clear  tracks  from  SPS-49  reports  only.  Tracking, 
using  MHT,  was  possible  whenever  the  SG-150  starts  reporting.  The  clustering  is  not  used 
here,  because  it  produces  tracks  with  lower  quality  than  MHT  without  clustering.  In  some 
situations,  it  can  even  not  produce  tracks  at  all. 

3.2.11  Run  12  &  13 

For  Run  12,  shown  on  Figure  11  (a),  aircraft  are  to  fly  a  straight  run  in  from  60 nm, 
overhead  ship  and  continue  out  to  60 nm.  During  Run  13  (see  Figure  11  (b)),  the  air¬ 
craft  initially  flies  towards  the  ship.  At  50 nm  from  the  ship,  it  alters  course  to  left  by 
approximately  45°.  After  30  —  45sec,  it  returns  to  heading  towards  ship.  When  20 nm  from 
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(a)  Scenario 


Figure  9:  Run  10 


ship,  it  alters  course  to  the  right  by  approximately  45°.  Again,  it  maintains  this  course  for 
30  —  45sec  and  then  resumes  a  course  which  over  tops  the  ship,  and  continues  outbound  to 
60  nm. 

Algorithm  behaviour 

The  system  is  able  to  track  all  targets  during  the  whole  run  using  JVC.  With  the  IMM  (3 
filters),  the  system  adds  some  false  alarms  into  the  track  at  the  end  of  the  run.  With  MHT, 
all  targets  are  tracked  during  the  whole  run.  This  concerns  the  clustered  MHT,  since  MHT 
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(a)  Scenario 


(b)  2D  Ground  Truth  (c)  3D  Ground  Truth 


Figure  10:  Run  11 
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Aircraft  2 


•  S80 

•  S546 

•  S80 2D 

•  S546  2D 


(c)  Ground  Truth  data 


Figure  11:  Runs  12  &  13 


without  clustering  did  not  work. 

3.3  Summary 

Table  8  summarizes  the  different  scenarios  performed  during  the  sea  trials  and  their  objec¬ 
tives.  Besides  runs  5,  6  and  10  that  are  not  analyzed  because  of  the  missing  ground  truth 
information,  runs  4  and  11  will  also  not  be  analyzed.  The  reason  is  that  the  computation 
of  the  association  MOPs  is  not  possible  due  to  the  data  incompleteness.  Decision  was  made 
not  to  include  the  results  of  the  estimation  MOPs  for  these  runs  as  well. 

In  most  runs,  the  implementation  of  MHT  in  CASE_ATTI  does  not  allow  tracking  when 
only  the  SPS-49  radar  system  reports.  MHT  worked  correctly  only  where  one  has  both 
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Parameters 

Values  JVC 

Value  MHT 

Maximum  Velocity 

500  m/s 

500  m/s 

Coordinate  Converter 

Standard  mode 

Standard  mode 

Gating 

Ellipsoidal  Gate 

Ellipsoidal  Gate 

Gate  Probability 

0.9999 

0.999 

Gate  Process  Noise 

500.0 

200.0 

Filter 

IMM  3 

IMM  3 

IMM  Process  Noise 

5,  250,  500 

5,100,200 

M  out  Of  N  System  Track 

3/12 

3/5 

Assignation 

JVCNN 

MHT  Clustering 

JVCNN  Process  Noise 

500.0 

MHT  Process  Noise 

200.0 

Mode  of  pruning 

Depth  control 

Number  of  Level  to  keep 

3 

Number  of  hypotheses 

200 

SPS49  Scan  RPM  &  Bearing  division 

12  &  4 

12  &  4 

SG150  Scan  RPM  &  Bearing  division 

60  &  32 

60  &  32 

Output 

Basic 

HSP 

Table  7:  Parameter  values  for  Runs  12  &:  13 


SPS-49  and  SG-150  reports.  Also,  the  clustering  within  MHT  does  not  necessarily  yield 
a  good  performance  for  most  of  the  runs.  In  a  few  runs  (1,  12&13)  the  use  of  clustering, 
in  combination  with  MTH,  provided  a  noticeable  benefit,  while  in  others  (runs  IB,  2,  3, 
4,  11)  clustering  resulted  in  lower  quality  tracks  or  prevented  tracking.  A  poor  clustering 
performance  increased  the  computation  burden  due  the  overhead  required  by  the  cluster 
management  (merge  and  split)  operations. 
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Run 

Objective 

Targets 

Type  of  Targets 

Range 

Altitude 

Speed 

01 

Separation 

S80,  S546 

Lear  Jet 

40  nrn 

4000  ft 

250  mph 

02 

Separation 

S80,  S546 

Lear  Jet 

40  nrn 

4000  ft 

200  mph 

03 

Accuracy 

S80,  S546 

Lear  Jet 

50  nrn 

4000  ft 

250  mph 

04 

Crossing 

S80,  S546 

Lear  Jet 

25  nm 

4000  ft 

250  mph 

05 

Merging 

Tl,  T2 

T-33 

25  nm 

4000  ft 

250  mph 

06 

Merging 

Tl,  T2 

T-33 

25  nm 

4000  ft 

250  mph 

07 

High  Dive 

Tl 

T-33 

20  nm 

2000  ft 

250  mph 

08 

Sharp  Turn 

T2 

T-33 

20  nm 

2000  ft 

250  mph 

09 

Radial 

Tl,  T2 

T-33 

20  nm 

4000  ft 

250  mph 

10 

Track  crossing 

and  maintaining 

ID 

S80,  S546, 
El,  E2,  E3 

2  Lear  Jets,  3  T-33 

40  nm 

250  mph 

11 

Track  crossing 

and  maintaining 
ID 

S80,  S546, 
El,  E2,  E3 

2  Lear  Jets,  3  T-33 

40  nm 

250  mph 

12 

SPS49/SG150 

Hand  Over 

S80 

Lear  Jet 

60  nm 

4000  ft 

250  mph 

13 

SPS49/SG150 

Hand  Over 

S546 

Lear  Jet 

60  nm 

4000  ft 

250  mph 

Table  8:  Summary  of  sea  trial  scenarios 
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4  Measures  of  Performance 


In  order  to  show  any  possible  advantage  (or  disadvantage)  of  the  Multiple  Hypothesis 
Tracking  (MHT)  association  algorithm  [4]  over  the  Jonker,  Volgenant  &  Castanon  (JVC) 
algorithm  [3],  a  set  of  Measures  of  Performance  (MOPs)  was  selected.  These  MOPs  will  be 
divided  into  two  main  classes,  namely,  the  association  MOPs  and  the  estimation  MOPs. 

Since  the  MHT  and  JVC  are  association  algorithms,  the  association  MOPs,  that  include 
Track  Purity  (TP),  Correct  Assignment  Ratio  (CAR),  Association  Correctness  (AC)  and 
Average  Completeness  Deviation  (ACD),  will  serve  for  the  primary  comparison.  The  esti¬ 
mation  MOPs,  that  include  the  Credibility  (Cr)  and  the  Error  Reduction  (Er),  will  only 
provide  an  insight  on  how  association  algorithms  affect  the  performance  of  the  estimation 
process. 

This  section  gives  the  list  of  the  MOPs  that  were  initially  used  for  the  comparison  of  the  two 
algorithms.  This  list  represents  a  sub-set  of  the  MOPs  used  for  sea  trial  data  analysis  [5]. 
Most  of  those  MOPs  are  based  on  the  confusion  matrix  that  is  defined  as  follows, 


Objects 

Oi 

o2 

Om 

C0 1 

C02 

Com 

T\ 

Cu 

C\2  ■■■ 

C\M 

Tracks 

T-2 

C-21 

C22 

C*2M 

tn 

Cm 

Cm  ■  ■  ■ 

Cnm 

where  the  elements  C\j  are  the  number  of  reports  originating  from  object  j  and  assigned 
to  track  i.  The  elements  Coj  are  the  reports  originating  from  object  j  and  not  assigned  to 
any  track. 

4.1  Track  Purity 

Track  Purity  assesses  the  percentage  of  correctly  associated  measurements  in  a  given  track, 
and  so  evaluates  the  association  (and  indirectly  the  tracking)  performance.  The  “purity” 
of  the  track  Tj  is  defined  as 


TP(tj) 


r  a. 


-1 


E  c>‘ 


max  Cji 

1  <i<a  J 


(4.1) 


where  Cji  is  the  number  of  the  reports  originating  from  the  ground  truth  platform  gl 
assigned  to  track  tj,  and  a  is  the  number  of  the  ground  truth  platforms  in  the  scenario.  The 
calculated  MOP,  in  this  work,  is  the  Weighted  Average  Track  Purity  (WATP),  a  statistic 
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of  Track  Purity  calculated  over  all  tracks  and  ground  truth  objects. 


TP 


EE0* 


b 

max  Cji 

3= 1 


where  b  is  the  number  of  tracks  in  the  system  database. 


(4.2) 


4.2  Correct  Assignment  Ratio 

Correct  Assignment  Ratio  measures  the  performance  for  a  ground  truth  platform  instead 
of  measuring  the  performance  for  a  track.  It  assesses  the  percentage  of  contacts  from  a 
ground  truth  platform  associated  with  the  correct  track.  The  Correct  Assignment  Ratio  of 
ground  truth  platform  gt  is  defined  as 


CAR(9i) 


Lj= i 


max  Ca 
l <i<b  J 


(4.3) 


where  Cji  is  the  number  of  the  reports  originating  from  the  ground  truth  platform  (ji 
assigned  to  the  track  tj,  and  b  is  the  number  of  the  tracks  generated  during  the  scenario. 
As  for  the  Track  Purity,  the  calculated  MOP,  in  this  work,  is  the  Weighted  Average  Correct 
Assignment  Ratio  (WACAR),  a  statistic  of  Correct  Assignment  Ratio  calculated  over  all 
tracks  and  ground  truth  objects. 


CAR  = 


where  a  is  the  number  of  the  ground  truth  platforms  presented  in  the  scenario. 


(4.4) 


4.3  Association  Correctness 


While  Track  Purity  and  Correct  Assignment  Ration  individually  may  reward  imperfect 
correlation  with  the  maximum  score  of  1.0,  their  geometric  mean,  which  will  be  referred  to 
as  Association  Correctness  (AC),  does  not.  The  Association  Correctness  is  defined  as 


AC  = 


1  1/2 

(TP)  (CAR) 

i  -1  r 


*  3 


i  V2  r 


EEC«  E  max  Cij  yy  max  Cij 


1  1/2 


(4.5) 

(4.6) 


It  reaches  the  maximum  value  of  1.0  if,  and  only  if,  each  row  and  each  column  of  the 
confusion  matrix  contains  exactly  one  nonzero  element,  thus  indicating  the  existence  of 
one-to-one  identification  between  the  tracks  and  the  ground  truth  objects,  that  is  a  perfect 
correlation.  Note  that  the  defined  metrics  does  not  consider  the  ambiguous  measurements 
(be.,  the  ambiguity  vector). 
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4.4  Average  Completeness  Deviation 


Instead  of  the  statistics  of  the  Completeness  History,  the  statistics  of  its  deviation  with 
respect  to  1.0  will  used.  The  Completeness  Deviation  is  thus  defined  as 


CD(t)  =  1  -  CH(t) 


(4.7) 


Also,  to  avoid  the  compensation  between  negative  and  positive  deviations2,  the  average  of 
the  absolute  values  is  computed,  as  follows 

ACD=±£\CD(t)\dt  (4.8) 

The  standard  deviation  (Completeness  Standard  Deviation)  is  computed  as  follows 


CSD 


1 

T 


1/2 


\CD{t)  -  ACD\2dt 


(4.9) 


4.5  Credibility 

Credibility  concerns  to  what  extend  one  can  trust  the  tracking/fusion  system  in  its  self- 
assessment  of  its  estimation  performance,  as  given  by  the  error  covariance  matrix.  There¬ 
fore,  an  estimator/fusion  system  is  said  credible  (or  consistent),  if  its  stated  level  of  per¬ 
formance  is  smaller  than  the  actual  one  [18].  The  Credibility  is  defined  here  by  the  two 
following  complementary  MOPs  [19].  Note  that  in  the  sequel  where  P  will  represent  the 
actual  covariance  matrix  and  P  its  estimate  provided  by  the  estimator /fusion  system. 

1.  Global  Credibility  Score:  defined  as 


GCS  =  min  <[  A,;  /  =  min  |  eigenvalue 


in  <f  ei( 


P-P 


(4.10) 


gives  a  sufficient  qualitative  condition  for  credibility  (or  non-credibility)  of  the  esti¬ 
mator.  If  GCS  >  0  the  system  is  credible,  otherwise  it  is  non-credible. 


2.  (Non)-Credibility  Factor:  defined  as  (if  the  estimator  is  credible) 
CF  =  max  /  Ajl  =  max  /  eigenvalue  ^ 


{ 

and  as  (if  the  estimator  is  not  credible) 

NCF  =  min  j^A*  j-  =  min  /  eigenvalue 
It  gives  a  quantitative  measure  of  the  level  of  credibility  (or  non-credibility). 


(4.11) 


P-P 


(4.12) 


'i.e.,  lack  of  clarity  and  lack  of  completeness. 
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4.6  Error  Reduction 


Error  Reduction  is  about  the  reduction/amplification  of  the  sensor  uncertainty  by  the  esti¬ 
mator.  It  uses  two  similar  MOPs  to  those  used  by  the  credibility  [6].  In  the  sequel,  z  will 
represent  the  measurement  vector  and  R  its  covariance  matrix,  while  £  is  the  part  of  the 
estimation  error  vector  x  corresponding  to  z,  and  T  its  covariance  matrix.  The  two  MOPs 
are  defined  as  follows. 


1.  Global  Error  Reduction  Score:  defined  as 


GERS  =  min 

i 


=  min 

i 


| eigenvalue 


r- ri ) 

~lT\j 


(4.13) 


gives  a  sufficient  qualitative  condition  for  sensor  error  reduction  (or  amplification) 
by  the  estimator.  If  GERS  >  0,  there  is  an  error  reduction  in  all  the  dimensions, 
otherwise  there  is  an  amplification  in  at  least  one  dimension. 

2.  Error  Reduction/ Amplification  Factors:  defined  as  (if  there  is  an  error  reduc¬ 
tion) 


M  ERF  =  min 

i 


=  min 

i 


| eigenvalue 


r- ri ) 


(4.14) 


gives  a  quantitative  measure  of  the  least  good  performance  (among  the  state  vector 
dimensions)  of  the  estimator  in  its  error  reduction.  MERF  stands  for  Minimal  Error 
Reduction  Factor.  When  there  is  error  amplification,  the  MOP  is  defined  as 


MEAF  =  max 

i 


=  max 

i 


eigenvalue 


R-rn 


(4.15) 


MEAF  stands  for  Maximum  Error  Amplification  Factor  and  gives  a  quantitative  mea¬ 
sure  of  the  worst  performance  (highest  error  amplification)  of  the  tracker. 


The  results  of  the  application  of  all  the  above  given  MOPs  to  the  comparison  of  the  JVC 
and  MHT  are  presented  and  discussed  in  Section  5. 
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5  Performance  evaluation 


This  section  gives  the  results  of  the  performance  evaluation  of  the  MHT  and  JVC  associ¬ 
ation  algorithms  within  CASE_ATTI  test-bed.  This  comparison  uses  the  set  of  the  MOPs 
presented  in  Section  4.  First,  a  summary  of  the  major  results  and  conclusions  will  be 
given  in  Section  5.1,  then  the  detailed  evaluation  of  the  two  systems  will  be  presented  in 
Section  5.2 

5.1  Summary 

An  important  conclusion  of  this  comparison  is  that  the  available  set  of  MOPs  does  not 
allow  stating  about  any  possible  improvement  in  both  association  and  estimation.  It  was 
noticed  that  when  association  performance  increases,  estimation  performance  decreases  and 
vice  versa.  This  problem  is  clearly  visible  on  the  results  of  Table  9.  There  is  no  run  that 
shows  a  superiority  of  one  approach  over  the  other  in  both  association  and  estimation.  The 
reasons  that  could  explain  thess  limitations  are: 

1.  the  independence  assumption,  between  the  association  and  estimation  operations, 
that  was  made  during  the  selection  of  the  MOPs.  It  is  however  obvious  that  the  two 
operations  are  tightly  dependent,  and  evaluating  one  without  considering  the  other 
is  senseless.  A  good  state  estimation  is  impossible  without  an  equally  good  contact- 
to-track  association  strategy,  and  vice  versa.  Inaccurate  tracks  give  an  inefficiently 
accurate  state  prediction,  that  lead  to  significant  errors  in  the  gating/association  of 
contacts  with  tracks. 

2.  the  type  of  analysis  applied  that  is  based  on  implicit  independence  assumption3  be¬ 
tween  association  and  update  phases,  and  which  uses  only  common  portions  of  the 
tracks  for  the  comparison.  Such  an  approach  penalizes,  in  terms  of  estimation  MOPs, 
the  solution/system  that  keeps  continuous  tracks  ( i.e .,  yields  a  good  performance 
in  association)  even  when  the  accuracy  decreases  below  the  contact  accuracy  (due 
mainly  to  manoeuvres).  The  reverse  is  also  true;  a  system  with  a  poor  performance 
in  association  will  produce  several  tracks  for  each  target,  with  an  accuracy  that  is 
higher  (at  least  equal  to  the  contact  one)  than  that  of  single  continuous  track.  This 
problem  was  noticed  is  several  situations  with  both  simulated  and  real  world  data. 

The  solution  to  this  problem  would  consist  of  using  the  whole  run  for  the  evaluation  of 
the  performance  and  finding  a  way  to  combine  association  MOPs,  particularly  the  track 
continuity  MOP,  with  the  estimation  MOPs  in  order  to  penalize  the  track  mis-association 
(discontinuity)  when  evaluating  the  track  accuracy.  Given  this  limitation,  it  would  not  be 
very  judicious  to  perform  an  in  depth  analysis  of  the  estimation  performance.  Therefore, 
the  analysis  will  be  only  performed  on  the  association  MOPS.  Nonetheless,  results  for  the 
evaluation  of  the  estimation  MOPs  will  be  given  for  completeness,  and  used  only  if  further 
insight  on  the  performance  of  the  algorithms  is  required. 

3  Given  the  nature  of  the  used  MOPs. 
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Since  each  run  has  its  particular  configuration,  such  as  the  number  of  targets,  the  type 
of  targets,  and  the  performed  manoeuvres,  none  of  the  two  algorithms  performs  well  for 
all  runs.  Table  9  shows  the  comparison  results,  in  term  of  association  and  estimation 
performance  for  each  run.  In  conflicting  situations,  more  importance  is  accorded  to  the 
association  performance. 


Run 

Association  Superiority 

Estimation  Superiority 

1 

= 

MHT 

lb 

= 

MHT 

2 

JVC 

MHT 

3 

MHT 

JVC 

7&8 

MHT 

JVC 

9 

= 

JVC 

12  &  13 

= 

JVC 

Table  9:  Comparison  of  MHT  and  JVC  algorithms  by  Run  for  all  MOPs 

It  is  clear  from  the  presented  results  that  there  is  no  real  advantage  of  one  approach  over 
the  other,  even  though  MHT  shows  a  slight  superiority  with  association  MOPs  and  JVC 
a  slight  superiority  in  estimation.  Note  that,  in  all  presented  tables,  bold  face  indicates 
a  noticeable  superiority,  while  normal  face  indicates  a  slight  superiority.  Details  of  the 
comparison  are  given  in  Section  5.2. 

The  same  conclusion  can  be  drawn  from  the  Table  10  that  shows,  for  all  runs,  the  aggregated 
results  of  each  MOPs.  While  the  MHT  is  superior  for  some  MOPs,  JVC  performs  better 
for  the  other  MOPs. 


MOP 

Superiority 

TP 

JVC 

CAR 

MHT 

AC 

MHT 

ACD 

JVC 

Cr 

MHT 

Er 

= 

Table  10:  Comparison  of  MHT  and  JVC  algorithms  by  MOP  for  all  runs 

5.2  Detailed  results 

This  section  presents  the  detailed  results  of  the  performance  evaluation  of  the  two  configu¬ 
rations  using  the  different  MOPs  presented  in  Section  4. 
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5.2.1  Association  MOPs 


Table  11  summarizes  the  results  of  this  evaluation  with  the  different  association  MOPs. 
Track  Purity  and  Correct  Assignment  Ratio  are  then  aggregated  therein  into  a  single  MOP, 
namely  the  Association  Correctness,  as  presented  in  Section  4.3.  From  association  per¬ 
spective,  the  analysis  boils  therefore  down  to  comparing  the  two  algorithms  using  only  two 
MOPs:  the  Association  Correctness  and  Average  Completeness  Deviation  (See  Section  4.4). 

The  results  of  Table  11  show  no  real  advantage  of  one  algorithm  over  the  other.  While 
JVC  performs  well  in  term  of  Average  Completeness  Deviation,  MHT  is  superior  in  the 
Association  Correctness. 


Run 

TP 

CAR 

AC 

ACD 

Superiority 

1 

JVC 

MHT 

MHT 

JVC 

= 

lb 

JVC 

MHT 

MHT 

JVC 

= 

2 

MHT 

JVC 

JVC 

JVC 

JVC 

3 

= 

MHT 

MHT 

= 

MHT 

7  &8 

MHT 

MHT 

MHT 

= 

MHT 

9 

JVC 

MHT 

MHT 

JVC 

= 

12  &  13 

JVC 

MHT 

MHT 

JVC 

= 

Table  11:  Association  MOPs 


Table  12  shows  the  scores  of  the  two  algorithms  for  the  Track  Purity,  for  each  run.  As 
previously  stated,  there  is  no  real  advantage  of  one  algorithm  over  the  other. 


Run 

JVC 

MHT 

MHT  with  Clusters 

Superiority 

1 

.903 

— 

.788 

JVC 

lb 

.901 

.896 

— 

JVC 

2 

.931 

.943 

— 

MHT 

3 

.998 

.998 

— 

= 

7  &  8 

.920 

.936 

.930 

MHT 

9 

.902 

.862 

— 

JVC 

12  &  13 

.925 

— 

.889 

JVC 

Table  12:  Track  Purity 

Table  13  shows  the  values  of  Correct  Assignment  Ratio  MOP  of  each  run,  where  MHT  seems 
to  be  superior  to  JVC  in  most  situations.  Table  14  shows  the  values  for  the  aggregated 
Association  Correctness  MOP.  MHT  maintains  its  superiority  due  to  its  higher  performance 
with  Correct  Assignment  Ratio. 

The  Average  Completeness  Deviation  MOP,  and  its  variance,  are  computed  for  the  different 
runs,  and  results  are  summarized  on  Table  15.  These  results  show  a  better  performance  of 
the  JVC  over  the  MHT. 


DRDC  Valcartier  TR  2003  -  287 


29 


Run 

JVC 

MHT 

MHT  with  Clusters 

Superiority 

1 

.278 

— 

.579 

MHT  Cl. 

lb 

.385 

.418 

— 

MHT 

2 

.413 

.334 

— 

JVC 

3 

.507 

.661 

— 

MHT 

7  &  8 

.411 

.487 

.920 

MHT  Cl. 

9 

.360 

.484 

— 

MHT 

12  &  13 

.275 

— 

.486 

MHT  Cl. 

Table  13:  Correct  Assignment  Ratio 


Run 

JVC 

MHT 

MHT  with  Clusters 

Superiority 

1 

0.501 

— 

0.676 

MHT  Cl. 

lb 

0.589 

0.612 

— 

MHT 

2 

0.620 

0.561 

— 

JVC 

3 

0.711 

0.813 

— 

MHT 

7  &  8 

0.615 

0.675 

.926 

MHT  Cl. 

9 

0.570 

0.646 

— 

MHT 

12  &  13 

0.505 

— 

0.657 

MHT  Cl. 

Table  14:  Association  Correctness 


Run 

JVC 

MHT 

MHT  with  Clusters 

Superiority 

1 

O.OI60.008 

— 

0.350o.o72 

JVC 

lb 

0.037o.oi7 

0.062o.o40 

— 

JVC 

2 

0.024o.o23 

0.469o.239 

— 

JVC 

3 

O.OOO0.000 

O.OOO0.000 

— 

= 

7  &8 

0.027o.o26 

0.025o.o24 

0.051o.o35 

= 

9 

0.020o.on 

0.139o.o57 

— 

JVC 

12  &  13 

0.043q.o20 

— 

0.151o.069 

JVC 

Table  15:  Completeness  History  (AC D variance) 


5.2.2  Estimation  MOPs 

Table  16  summarizes  the  results  for  two  estimation  MOP,  namely  the  Credibility  (see  Sec¬ 
tion  4.5  for  definition)  and  the  Error  Reduction  (see  Section  4.6  for  definition).  The  objective 
here  is  not  an  in  depth  comparison  of  the  two  approaches.  Since  MHT  and  JVC  are  associ¬ 
ation  algorithms,  this  comparison  aims  at  showing  how  the  performance  of  the  association 
algorithm  affects  the  estimation  one.  From  this  perspective,  JVC  shows  marginally  better 
behaviour. 

Table  17  gives  the  percentage  of  time  during  which  the  system  was  credible  in  its  self- 
assessment  of  the  tracking  performance.  This  MOP  is  only  based  on  GCS  MOP.  To  better 
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Run 

Target 

Credibility 

Error  Reduction 

Superiority 

1 

S80 

JVC 

EE 

JVC 

S546 

MHT 

MHT 

MHT 

IB 

S80 

MHT 

EE 

MHT 

S546 

MHT 

= 

MHT 

2 

S80 

JVC 

MHT 

MHT 

S546 

MHT 

MHT 

MHT 

3 

S80 

JVC 

JVC 

JVC 

S546 

= 

= 

= 

7  &8 

T1 

EE 

EE 

= 

T2 

= 

JVC 

JVC 

9 

T1 

EE 

EE 

EE 

T2 

= 

JVC 

JVC 

12&13 

S80 

JVC 

JVC 

JVC 

S546 

MHT 

JVC 

JVC 

Table  16:  Comparison  for  estimation  MOPs 


show  the  CF  and  NCF  notions,  color  bars  (respectively  green  and  red)  are  used  on  the 
presented  graphics.  According  to  the  scores  of  Table  17,  none  the  two  algorithms  yields  a 
good  performance  in  term  of  credibility. 

Similarly,  Table  18  gives  the  percentage  of  time  during  which  there  was  a  reduction  of  the 
sensor  uncertainty  by  the  tracking/fusion  system.  As  for  the  Credibility,  Table  18  does  not 
reveal  any  good  performance  from  the  two  approaches  in  terms  of  error  reduction.  Since 
the  values  represent  the  percentage  of  time  during  which  there  is  a  reduction  of  uncertainty, 
a  low  value  means  the  system  are  not  reducing  error  during  most  of  time.  As  shown  on 
Table  18,  MHT  and  JVC  obtain  approximately  the  same  level  of  values  in  most  of  the  cases, 
the  differences  are  not  more  than  2%  or  3%. 
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Credibility  MOPs 


xIO5  xIO4 


Time  Time 


(c)  MHT  -  S80 


(d)  MHT  -  S546 


Figure  12:  Credibility  for  Run  1 
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Credibility  MOPs  Credibility  MOPs 


iitf  » 


600  650  700  750 


850  900  950 


(c)  MHT  -  S80 


(d)  MHT  -  S546 


Figure  13:  Credibility  for  Run  IB 
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Credibility  MOPs  Credibility  MOPs 


(c)  MHT  -  S80 


(d)  MHT  -  S546 


Figure  14:  Credibility  for  Run  2 
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Credibility  MOPs  Credibility  MOPs 


xIO4  xIO4 


1300  1400  1500  1600  1700  1800  1200  1300  1400  1500  1600  1700  1800 

Time  Time 


(a)  JVC  -  S80 


(b)  JVC  -  S546 


x  104 


x  104 


(d)  MHT  -  S546 


Figure  15:  Credibility  for  Run  3 
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Credibility  MOPs  Credibility  MOPs  Credibility  MOPs 


(c)  MHT  -  T1 


(d)  MHT  -  T2 


xIO4  xIO4 


Time  Time 


(e)  MHT  (with  clustering)  -  T1 


(f)  MHT  (with  clustering)  -  T2 


Figure  16:  Credibility  for  Runs  7  &  8 
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Credibility  MOPs  Credibility  MOPs 


(c)  MHT  -  T1 


(d)  MHT  -  T2 


Figure  17:  Credibility  for  Run  9 
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Credibility  MOPS  Credibility  MOPs 


(a)  JVC  -  S80 


(b)  JVC  -  S546 


(c)  MHT  -  S80 


(d)  MHT  -  S546 


Figure  18:  Credibility  for  Runs  12  &  13 
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(c)  MHT  -  S80 


(d)  MHT  -  S546 


Figure  19:  Error  Reduction  for  Run  1 
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x  104 


(c)  MHT  -  S80 


(d)  MHT  -  S546 


Figure  20:  Error  Reduction  for  Run  IB 
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xIO4  xIO4 


Figure  21:  Error  Reduction  for  Run  2 
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(d)  MHT  -  S546 


Figure  22:  Error  Reduction  for  Run  3 
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(e)  MHT  (with  clustering)  -  T1 


(f)  MHT  (with  clustering)  -  T2 


Figure  23:  Error  Reduction  for  Runs  7  &  8 
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Figure  24:  Error  Reduction  for  Run  9 
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Run 

Target 

JVC 

MHT 

Superiority 

1 

S80 

3.72 

1.18 

JVC 

S546 

19.74 

86.06 

MHT 

IB 

S80 

0.0 

1.63 

MHT 

S546 

0.29 

12.21 

MHT 

2 

S80 

9.87 

7.10 

JVC 

S546 

16.15 

39.87 

MHT 

3 

S80 

14.42 

0.82 

JVC 

S546 

19.06 

18.91 

= 

4 

S80 

16.18 

22.87 

MHT 

S546 

90.12 

90.50 

= 

7  &8 

T1 

91.08 

89.12 

= 

T2 

5.70 

4.53 

= 

9 

T1 

4.84 

4.84 

= 

T2 

9.66 

9.80 

= 

11 

S80 

35.29 

22.79 

JVC 

S546 

0.41 

38.15 

MHT 

El 

0.0 

1.62 

MHT 

E2 

30.87 

23.86 

JVC 

E3 

17.63 

18.08 

= 

12&13 

S80 

2.28 

0.67 

JVC 

S546 

0.0 

1.23 

MHT 

Table  17:  Percentage  Time  of  Credibility 
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Run 

Target 

JVC 

MHT 

Superiority 

1 

S80 

10.82 

11.82 

EE 

S546 

24.88 

100.0 

MHT 

IB 

S80 

54.41 

53.69 

EE 

S546 

72.79 

72.51 

= 

2 

S80 

38.23 

58.89 

MHT 

S546 

29.94 

81.94 

MHT 

3 

S80 

26.66 

15.13 

JVC 

S546 

22.89 

22.71 

= 

4 

S80 

51.40 

52.00 

EE 

S546 

93.03 

93.17 

= 

7  &8 

T1 

95.64 

95.84 

= 

T2 

26.91 

21.12 

JVC 

9 

T1 

8.23 

8.33 

EE 

T2 

21.82 

10.79 

JVC 

11 

S80 

39.98 

35.33 

JVC 

S546 

36.84 

43.89 

MHT 

El 

0.0 

8.63 

MHT 

E2 

32.43 

32.45 

EE 

E3 

26.17 

23.95 

JVC 

12&13 

S80 

76.11 

40.50 

JVC 

S546 

20.77 

2.63 

JVC 

Table  18:  Percentage  of  Time  of  Error  Reduction 
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(c)  MHT  -  S80 


(d)  MHT  -  S546 


Figure  25:  Error  Reduction  for  Runs  12  &  13 


DRDC  Valcartier  TR  2003 


287 


47 


This  page  intentionally  left  blank. 


48 


DRDC  Valcartier  TR  2003  -  287 


6  Conclusion 


This  report  presents  the  results  of  the  work  performed,  at  DRDC  Valcartier,  as  a  part  of 
the  performance  evaluation  of  COMDAT  MSDF  technology.  A  previously  performed  data 
analysis  showed  a  limited  performance  of  COMDAT  MSDF  with  respect  to  the  association. 
This  work  aims  at  evaluating  a  candidate  alternative  to  the  JVC  algorithm  used  by  MSDF, 
namely  the  MHT,  a  equivalent  implementation  of  which  is  available  in  CASE_ATTI  test¬ 
bed.  The  latter  is  used  as  a  comparison  environment.  This  comparison  uses  a  limited  set  of 
MOPs  and  was  conducted  under  time  and  resources  constraints,  during  the  June-August, 
2003  time  frame.  Therefore,  the  presented  results  and  the  following  conclusions  should  be 
considered  in  the  light  of  those  constraints. 

1.  Even  though  it  is  proved  theoretically,  and  using  synthetic  data,  that  the  MHT  offers 
an  optimal  solution  to  the  association  algorithm,  the  presented  results  show  no  real 
advantage,  with  real  world  data,  of  MHT  over  JVC. 

2.  Implementation,  optimization  (of  the  code),  and  the  parameter  tuning  of  the  MHT 
algorithm  is  costly  (both  in  terms  of  time  and  money).  These  results  are  optimized 
to  the  extent  that  each  algorithm  was  tuned  for  each  run.  If  parameters  had  been 
tuned  to  some  nominal  values  and  then  held  constant  over  all  runs  then  the  resulting 
performance  may  be  worse,  and  comparison  of  the  two  algorithms  may  be  at  the 
advantage  of  the  JVC,  since  it  is  less  sensitive  to  the  parameter  tuning. 

3.  Behaviour  of  the  MHT  with  real  world  data  is  still  not  well  understood.  More  sen¬ 
sitivity  analysis  is  required  to  gain  a  clear  idea  of  how  the  MHT  could  work  in  a 
practice.  A  badly  coded/tuned/used  MHT  may  yield  worse  results  than  the  simple 
Nearest  Neighbour  algorithm.  This  was  the  case  with  MHT  when  compared  with  the 
JVC  that  is  easier  to  tune. 

4.  Rather  than  investigating/envisaging  the  implementation  of  new  complex  algorithms 
in  COMDAT  MSDF,  it  would  be  more  judicious  and  safer  to  understand  why  the 
JVC  did  not  yield  the  expected  performance  with  real  world  data.  A  better  tuning 
might  be  required. 

5.  The  improvement  of  other  functionalities,  such  the  gating  and  the  estimation,  may 
help  increase  the  performance  of  the  association  algorithm.  The  use  of  a  different 
metrics  may  be  used  to  improve  the  gating  performance,  and  therefore  the  association. 

It  important  to  notice  that  this  work  represents  an  initial  investigation  that  leaves  a  number 
of  questions  unanswered.  These  questions  are  being  addressed  by  COMDAT  TDP  team, 
mainly  at  DRDC  Atlantic. 
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